Lecture 1: Planning an Analysis

Max Griswold

Pardee RAND Graduate School

1/1/24

The purpose of this class

  • Most researchers learn to conduct quantitative analyses through apprenticeship.
  • However, we can do better analyses if we follow a set of building blocks and interrogate our research designs.
  • The purpose of this class is to learn skills to improve our analyses and to practice doing an analysis of our own.

Learning objectives

By the end of the quarter, you should be able to:

  • Evaluate the strength and weaknesses of a quantitative research design.
  • Provide effective feedback and comments to other researchers on their research design.
  • Implement a exploratory or causal analysis using R code.

Course format

Lecture days:

  • Learn skills and resources to execute a research design.
  • Learn how to use R to conduct an analysis.

Workshop days:

  • Share our code in small groups.
  • Present results from interim work.
  • Discuss strengths and weakness of published papers.

Class resources

Class Github page

Gelman, Hill, and Vehtari, 2022

Blair, Coppock, and Humphreys, 2023

Planning an analysis

Robustness: 
Will my research design provide a reliable answer?

Feasability: 
Is my research question answerable given my constraints?

Relevancy: 
Is this a meaningful question to ask and who is the audience?



Relevancy is particularly important for policy analysis!

Robustness

  • An analysis is only useful if the results are reliable (unbiased and precise).
  • Our research design determines how much trust we can place in the estimates we develop.
  • To determine if our design is robust, we start with a theory of how the world is working.
  • For an exploratory analysis, we need to use theory to think critically about measurement. For a causal analysis, we also need to think about the research design.

Robustness - Generating a theory

Causal pathway diagrams allow us to visualize our theory and begin to develop a design plan.

  • We want to know how D effects Y.
  • We might also be interested in how M changes the effect of D.
  • We need to remove X from our model.
  • We need to make sure we don’t condition on K.

Robustness - Generating a theory

Robustness - Generating a theory

Robustness - Testing a theory

Using our theory, we can try to determine an accurate data strategy (measurement) and an answer strategy (design) to test our research question.

Feasibility

  • Our research questions need to be scoped so we can answer them given available time and funding.
  • To determine if our question is feasible, we need to use project management tools to evaluate if our time and resources will be enough to meet the needs of the project.
  • Alternatively, we can scope our research question to fit our available resources

Feasibility - Plan out the steps

Try working back from then end of the study (the estimate) to the necessary methods and data needed to obtain that result.

Feasibility - Plan out the steps

Flowcharts can be used to plan out analysis steps and also serve as a useful rhetorical tool.

Feasibility - Plan out the steps

Feasibility - Planning out time

Gantt charts provide an intuitive way to plan out timing.

Feasibility - Planning out budget

Use a a budget estimator adapted to your schedule and products.

Feasibility - Scoping questions

If you think about it, specificity is also an act of shrewdness. A survey of 40 years of literature is vulnerable to all kinds of objections

The subject of “Gealogy,” for instance, is much too broad a topic. “Vulcanology” as a branch of geology, is still too comprehensive. “Volcanoes in Mexico” might be developed into a good but superficial paper. However, a further limitation to “The History of Popocateptl” (which one of Cortéz’s conquistadores probably climbed in 1519 and which erupted violently as late as 1702) would make for a more valuable study. Another limited topic, spanning fewer years, would be “The Birth and Apparent Death of Paricutin”

Here, I would suggest the last topic, but only if the [PhD] candidate really says all there is to say about that damned volcano.

-Eco, 1977

  • Limit the geography or time-periods of the study.
  • Study less outcomes.
  • Use a simpler method.
  • Stratify fewer variables.

Relevancy - Generating questions

Figuring out impactful research questions seems to be an art; here’s some starting points I’ve found useful:

  • Be an informed citizen and policy observer.
  • Look out for methodological shortcomings in previous work.
  • Ask for feedback on ideas constantly (and not just from researchers)

Relevancy - Generating impact

Think critically about who will care about the results and how the results will be used:

  • Who is the audience for the study’s results?
  • How will a decision-maker use these results?
  • What are the policy-levers available to the decision-maker?

Relevancy - Disseminating results

Your results are only useful if they get to the right audience. Start thinking about how your results will be disseminated at the beginning of your project!

  • Start writing up your research article at the beginning. Good writing takes time.
  • Think through how you’ll describe results if the study’s hypothesis is confirmed. But also think through how the results will be useful if the hypothesis is rejected.
  • Set up multiple plans to disseminate results beyond a research article.

Next class

For class on Wednesday, please prepare the following:

  • A short elevator-pitch for your research
    • No longer than 1-2 minutes.
    • What is your research question?
    • What data will you use to answer your question?
    • What research design do you plan to use?
    • Who will use the results of your analysis?
  • Read the Tong, 2019 paper.
    • What arguments did you find convincing/unconvincing and why?
  • Download Rstudio and GitHub desktop.
    • You might need to ask IS for permission to install these programs.
    • Try loading your data into R and begin investigating your dataset.

Pardee RAND